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ABSTRACT 

Basic issues in workplace education evaluation can be 
structured into five discussion questions. The first is who wants to 
know what information for what purpose. The "who" constitutes the 
program's stakeholders both at and beyond the program level. Further 
inquiry is then needed to determine what kind of information these 
stakeholders typically want from an evaluation and purposes for which 
they might use the information. The second question concerns the 
goals of workplace education and the indicators of progress toward 
those goals. All stakeholders' expectations for the program must be 
clarified. Through airing, negotiating, and prioritizing these 
expectations, agreement may be reached on what the program should be 
trying to achieve and on which of those program goals to focus most 
in the evaluation. Indicators of progress or success or ability 
should be determined so evaluators know what specific evidence to 
look for. The third question is about determining which are the other 
kinds of information on which an evaluation might focus. The fourth 
question concerns what sources of information exist and how 
information can be gathered from those human and material sources. 
Data-gathering activities may include interviews, questionnaires, 
observation, tests, and review of records. The fifth question is on 
what can be done to improve how evaluation is used. Suggestions are: 
assessing strengths and limitations of evaluation options, 
demystifying evaluation, and making a personal investment. (YLB) 
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Introduction 



(Oetting thg "lav nf rtio land" 

In the last 2 1/2 years, I have focused on the question of how 
to most appropriately evaluate workplace literacy programs. I have 
approached this work from what I considered a participatory, 
collaborative perspective. Such an approach recognizes that a 
literacy program is composed of a number of different stakeholders, 
each of whom brings particular interests and resources to the effort. 
In a collaborative effort, each stakeholder has opportunities to shape 
the program to build on his/her resources and meet his/her 
interests. 

As I got into the question of how best to evaluate workplace 
education programs, I saw the need to more fully understand the 
"lay of the land" of how evaluation is currently done in workplace 
basic skills programs. Through direct interaction with workplace 
programs, talking with others involved in the field, and reading, I 
now have a better grasp of the state of the art I have clarified basic 
things like how evaluation is now done, who is involved, why 
evaluation is done, for whom it is done, the data-gathering tools 
people use, and how people think of evaluation in the context of 
workplace education programs. 

In the process, I have identified some valuable resources, as 
well as some problems which need to be addressed by those 
interested in improving how evaluation is done in workplace 
contexts. 

For me personally, taking the time to learn what others are 
doing and thinking has been both rewarding and at times frustrating. 
It is rewarding because I now feel that I have a stronger foundation 
for the work of developing more effective ways of approaching 
evaluation. It is also frustrating because this notion of workplace 
literacy evaluation is a new one for me and the others I work with. 
We thus often lack a common language, common expectations, 
common prior experience. This can result in confusion. 

Recent research and development projects: 

This morning I'd like to share some findings from a number of 
projects I've worked with in the past 2 1/2 years: 



• ThA National ft isrjtnte for literacy has funded a one-year 
research project in which my colleague Laura Sperazi and I are 
field-testing a team-based approach to evaluation. We first began 
developing this model two years ago for the Massachusetts 
Workplace figuration Initiative. 

• arc CANADA, a technical assistance organization based in 
Toronto which specializes in workplace education issues, is 
developing a similar team-based planning and evaluation model 
for use in Canada. 

• The New York State Educati on Department has, in the past 1 1/2 
years, conducted workshops for workplace educators and 
redesigned its funding guidelines, with the aim of reshaping how 
workplace programs plan and evaluate themselves. 

• And here in Texas, I've had the opportunity to learn how 
planning, evaluation, and assessment are used by El Paso 
Co mr™™^ College' s program for garment workers, m the 
WriftfrmT ] m fnrtional Network program for small businesses in 
San Marcos, and by other practitioners whom I interacted with m 
workshops at last year's Texas Workforce literacy Conference. 

Why taking time to study curr ent practice is important: 

It is important for workplace education practitioners and policy 
rnakerstotakethetimetoMomeirhomeork"andfammarize 

themselves with what people are doing in this important area of 
evaluation. If they don't, they tend to rush into setting up evaluation 
practice and policy without knowing useful tools they might use and 
the stumbling blocks they might run into. 

To facilitate our understanding of some basic issues in this 
area, I've structured this discussion today around five discussion 
questions: 

1. In workplace education evaluations, wha wants to know wjiai 
information for what DUTPQSf? 

2. What are the goals of workplace education? And what are 
indicators of progress toward those goals? 

3. Besides determining whether goals are being met, what offiex 
kinds of information might an evaluation focus on? 



4. What are possible sources of information? And how can 
information be gathered from those sources? 

5. What can we do to improve how we use evaluation? 

Question #1: In workplace education evaluations, 
who wants to know what information for what 
purpose? 

Commonly, when people talk about evaluation they 
immediately get into a debate about which "test" to use. This focus 
on which test to use is understandable but unfortunate, because tests 
are only one of many possible ways of gathering information for use 
in an evaluation. 

And in many cases, tests as most people normally think of 
them (that is, "standardized" tests) might not be appropriate at all, 
because: 

• Standardized tests often don't get the information really 
desired by the stakeholders in a workplace context 

• These tests can have the negative effect of turning off 
participants, not only to the evaluation process but to the 
education program itself. This is because tests often have 
negative, punitive, or threatening connotations in many people's 
minds. 

When looking at the question of how to set up a program 
evaluation, I have found that a useful place to start is a basic 
question of "Who wants to know what information for what 
purpose?" (This is question I learned from my advisor at the 
University of Massachusetts, David Kinsey.) If you and the people 
you are planning an evaluation with can answer that question, you 
have laid the foundation for a useful, meaningful evaluation. 

1. a. Who are th e stakeholders in workplace literacy 
programs? 

When we look at evaluations now being prepared in workplace 
programs nationally, the potential "who's" in this question constitute 
the program's "stakeholders" or " interest groups." These are the 



people who - for various reasons - have an actual or potential 
"interest," "stake," or "investment" in workplace literacy efforts. 

Based on your own experience, who would you consider to be 
the stakeholders in workplace literacy efforts in this country? 
(Author's note: The participants at this point developed the following 
list of stakeholders, which were recorded on a flipchart:) 

Potential stakeholders in workplace literacy programs 
(as defined by Dallas conference participants) 

• Students 

• Management 

• Teachers 

• The company's human resources department 

• Funders 

• OSHA 

• Taxpayers 

• Employees not in class 

• Insurers 

• Administration of the educational institution 

• Union 

• Families of students 

• The company's stockholders 

• Other local companies 

• The company's customers 

• The community and larger society 

Your list in fact reflects what most documents on workplace 
literacy now tell us. They say that workplace literacy programs 
should be seen as a "partnership" among a variety of stakeholders or 
interest groups. Typically these partners are listed as "management," 
"workers," "unions," and the "education provider." More recently, 
"learners"- the workers participating in the program - have also 
been added to some people's lists of stakeholders. 

But this kind of depiction of a workplace program's 
stakeholders is a bit of an over-simplification, because it leads one to 
believe that these are monolithic categories. 

• In fact, "management" needs to be further broken down to 
include not only higher-level managers, but also supervisory-level 
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people, as well as management representatives from various 
departments within an institution. 

• This is also true of learners themselves. They: 

- come from different departments and communities, 

- have had different experiences with literacy, and 

- can't be treated as though they are all the same and have 
identical interests and knowledge bases. 

Being more precise about who we consider to be a program's 
stakeholders is not being "picky" or "academic." If this clarification 
of who interests groups are isn't done, program planners are likely to 
focus only on serving one or two sets of interests while leaving other 
interests out 

When this happens, those whose interests are not being 
responded to are likely to see the program as of limited relevance, 
not get much out of it, and not give much to it in terms of support 
and cooperation. 

Similarly, planners of evaluations need to be clear just who 
might constitute "audiences" for the information to be generated in 
the evaluation. A good evaluation would focus on developing 
information relevant to the people who are asking for the evaluation. 
A clear list of potential stakeholders is a good place to start when 
trying to clarify the audiences for an evaluation. 

A comprehensive list of potential stakeholders in workplace 
education programs might consist of: 

POTENTIAL STAKEHOLDERS 
IN 

WORKPLACE EDUCATION PROGRAMS 

At the program level 

Management representatives (from various departments 
concerned about basic skills) 

• Higher-level managers 

• Supervisory-level managers 



Union representatives 



Employees: 

• Current participants in the program 

• Possible participants in the program 

• Employees who chose not to participate in the program 

• Co-workers not likely to participate in the program. 

Education staff (from inside or outside the workplace) 

• Instructors 

• Curriculum designers 

• Administrators 

Learners' "communities" 

• Family members 

• Friends 

• Other local community institutions 

Tfie nrgani7ation's c ustomers 

• Customers/clients who want quality service and products 
from the organization. 

Beyond the program level 



• Public sector funders (local, state, and federal) 

• Private-sector funders (e.g., unions, trade associations) 

Economic development polirv makers 

• Public sector (e.g., state workforce development agencies....) 

• Private sector (e.g., Hudson Institute, Southport Institute...) 

Adult ed ucation providers 

News media with responsibility for covering workforce and 
education issues 

Vendors of workplace education services and products 

A truly comprehensive list of stakeholders might be some 
combination of the above two lists and others like it developed by 
others in the field. 



Question #l.b.: What kind of information do these 
stakeholders typically want from an evaluation? 

If we see evaluation as to a large degree a process of gathering 
information for someone to use to make decisions with, then we need 
to be clear just what information it is we should be gathering. This 
brings us back to our question of "who wants to know what 
information for what purpose?" 

If we agree that the above lists of stakeholders represent the 
range of possible audiences for an evaluation, we next need to ask 
just what kind of information it is that these groups want an 
evaluation to provide them with. Let's just look at our lists of 
stakeholders for a minute, and take them one by one. 

To make this more concrete, let's assume, for example, we have 
been asked to plan an evaluation of a workplace education program 
for nurses aides in a nursing home. Management wants to help aides 
to be prepared for more-responsible roles in a new patient care 
system which emphasizes teamwork, "quality," new safety 
procedures, and careful written documentation. 

We have been told that one of the audiences we should develop 
information for is the category of "supervisors." These are the people 
who decide whether to release the employee to go to class and can 
otherwise encourage - or discourage-the employee's participation. 
Supervisors can also provide valuable ideas for how to make a 
curriculum meaningful. 

If one of our audiences were to be supervisors, what kind of 
information do you think they would *ike us to provide them with? 
(Author's note: The paricipants at this point developed the following 
list:) 

What [nfo^ation Might Supervisors Want? 
(as defined by Dallas conference participants) 

• Are residents more independent? 

• Is communication better between aides and supervisors? 

• Is documentation better? 

• Are there fewer "incidents" (complaints)? 
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• How much employee time was required (to participate in the 
education program)? 

In an actual nursing home program I worked with, supervisors 
referred to a list of skills which all nurses aides were supposed to 
possess. This list, developed by the state nursing home association, 
included: "using oral and written English to report emergencies, fill 
out flow charts, communicate with patients at bath and meal times, 
read dietary information . . ." ) 

If we look at another category of "stakeholders" in this nursing 
home program - "learners" - what information might program 
participants want to get from an evaluation? (Author's note: Here 
participants developed the following list) 

Wh a t Information Might Learne rs Want? 
(as defined by Dallas conference participants) 

• Did my test scores go up? 

• Will I get a raise or promotion as a result of participating? 

• What did supervisors say about us? 

• What did residents say about us? 

• What more do we have to learn? 

• Will I be fired if I don't improve my skills? 

• What did the evaluation tell my supervisor about me? 

Let's look at one more category of stakeholders: "funders." In 
this case, let's assume that a key funder is the U.S. Department of 
Education. What information might the U.S. Department of Education 
want from an evaluation of this program? (Author's note: Here the 
participants developed the following list:) 

What Information Might the U.S. LVnt. Of Ed. W ant? 
(as defined by Dallas conference participants) 

• Did the program do what they said they would do? 

• Did the program document impact on productivity? 

• Is the program replicable by other providers elsewhere? 

In addition to the above kinds of information, U.S. Department 
of Education might want to know whether it should continue funding 
this program, as well as what worked and what didn't (so we know 
how to shape policy in the future). 

i%) 
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In my own interaction with the field, I have found that — 
when stakeholders are asked what types of information they want 
from an evaluation, they respond in several ways, as follows: 

TYPES OF INFORMATION 
COMMONLY WANTED BY STAKEHOLDERS 

1 . What are/should be the goals of the program? 

2. What in fact is being achieved in the program? 

a. In terms of the intended/anticipated outcomes ? 

b. In terms of other, unanticipated outcomes? 

3. What factors are helping or hinripring progress toward program 
goals? 

4. Should we continue to "invest" (money and time) in the 
program? 

5. If it is to be continued, what needs to be done (and by whom) 
to improve the program? 

Question #l.c : What do stakeholders want to use 
that information for? 

We have now clarified, in broad terms, what kind of 
information the various stakeholders in workplace literacy programs 
might like from an evaluation. Imbedded in those categories of 
information are answers for the third part of our question "who 
wants to know what information for what purpose?" If we look at 
the list of "types of information commonly wanted," the following are 
purposes which stakeholders might use the information for: 

COMMON PURPOSES FOR EVALUATING 
WORKPLACE EDUCATION PROGRAMS 

To help stakehold ers decide: 

• whether to continue investing in the program and, if so, 

• how those "investments" (resources) should be used. 
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Question #2: What are the goals of workplace 
education? And what are indicators of progress 
toward those goals? 

2.a. What do stakeholders see as the goals of 
workplace education? 

As I mentioned, many stakeholders in workplace programs 
want their evaluations to clarify what in fact the program should be 
trying to achieve. In other words, they see evaluation as a means for 
goal clarification. It is important for stakeholders to have a clear 
understanding of the program's goals or purposes for two reasons: 

1, If stakeholders understand the purposes of the program, they 
can know how best to tailor services to meet those interests. 

2. With a clear statement of goals, those designing an evaluation 
will know what criteria to use to judge whether in fact the 
program is meeting its goals. 

Put another way, if you're planning to start a week-long car 
trip, it's important to know your destination so that: 

1. you'll be able to figure out how to get there, and 

2. you'll know whether you made it when the week is up. 

When you read guidelines from some funders, research reports 
from some researchers, and press accounts of the workplace literacy 
issue, you might conclude that there is only one legitimate purpose 
for workplace literacy programs. That purpose is "to improve 
employee productivity (or job performance)." Some would even go 
so far as to speak for the entire U.S. business community and say 
that this is in fact the view of all employers and that, therefore, 
workplace literacy education is fundamentally a "bottom line" issue 
of improving the productivity of the U.S. workforce. 

However, when you listen to what other people are saying 
about this topic, it turns out that there is not universal agreement 
within the field on this question of "What are the purposes of 
workplace literacy education?" These "people," by the way, are the 
full range of stakeholders: employers, union representatives, 
learners, and education providers. When these stakeholders are 
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asked that question, they cite "improved job performance" as well as 
a number of other purposes. 

" WHAT ARE THE PURPOSES OF WORKPLACE LITERACY 

EDUCATION? ": 
WHAT PEOPLE ARE SAYING 

w C™nfiXT" a1 *7fflr Orientation: 

To improve workers' job performance 

• Workers are better able to perform specific tasks within 
specific jobs. 

• Workers are better able to work in teams to analyze and 
solve problems across tasks and jobs. 

To encourage employee's personal development 

• Workers improve their job opportunities. 

• Workers can achieve non-job-related goals. 

To facilitate both improved job performance aiLdL 
personal development 

"DprnntPYMialiypH" orientation: 

To help workers develop "basic skills" for use at a 
later date 

We need to be careful about jumping to conclusions about the 
purposes which various stakeholders expect workplace education 
programs to achieve. For example, it is tempting to assume that all 
employers take the "job performance" view, while employees and 
unions take the "personal development" perspective. This, however, 
is in fact not true. For example: 

• Some employers conceive programs to be primarily for 
employees' "personal development" 

• Many employers see these programs as a mix of job-related and 
personal growth. 

• Many educators appear not to support a "contextualized" view of 
learning (perhaps due to their historical reliance on 
decontextualized learning in school settings). They therefore 
prefer a "decontextualized" view. 
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• Some educators assume that employers and funders want 
improved job-performance. The educators adopt job-performance 
"lingo" and goals, even if they don't fully believe in them. 

The point here is that, when planning an evaluation, it is very 
important to take the time to clarify what all stakeholders* 
expectations for the program are. It might not be possible for a 
program to meet all the expectations of all stakeholders, but these 
expectations should be aired, negotiated, and prioritized, hi so doing, 
you can come to some agreement on what the program should be 
trying to achieve and which of those program goals to focus on most 
in the evaluation. 

While you might think that all stakeholders are dear about 
what the program should be achieving, in fact many aren't so sure. 
This could be due to several reasons: 

WHY STAKEHOLDERS AREN'T ALWAYS CLEAR 
ABOUT PROGRAM GOALS 

• Fffl frgholflprs were n o t available to give input at the beginn i ng : 
In some cases, stakeholders come into the program cycle late, and 
were never involved in the goal-setting done early in the 
program. 

• Stakeholders were avail a ble , but no one asked for the i r input : 
In some cases, stakeholders were in fact available during the 
early planning of the program, but they were never given the 
opportunity to have input or learn just what the goals were that 
were being developed by others. 

m flffifre holders are n o t familiar with ranee of Possible PUTPQSeS 
for workplace education : Most stakeholders are "new" to the idea 
of basic skills education in the workplace. They have not thought 
or studied much about what such programs might achieve. 
Stakeholders might thus figure the program should teach the 
kinds of "basic skills" they learned in grammar school. Or they 
might merely adopt the language of funders or media accounts 
which emphasize literacy for "the bottom line." 

> fr eeholders are in timidated from stating their real hopes and 
expectations: Some stakeholders might think that a program 
should focus on job-related goals. However, they might be 
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reluctant to say so if everyone else involved in the program is 
talking about the a program as a vehicle for the employees' 
personal growth. And vice versa. 

• KPfliiriPs havp changed and goals haven't been updated: In 
some cases, the context and/or workforce have changed since the 
beginning of the program, and goals have not been updated to 
reflect those changes. 

• Goals are stated in overly-broad, intangihte terms; In some 
cases, goals were set at the beginning of the program, but they 
were not stated in specific terms. (For example, program planners 
might have declared that they wanted to "improve 
communication" or "improve productivity" through the education 
program, but they never got any more specific than that.) In such 
cases, stakeholders often "go with the flow" but are never quite 
sure in tangible, practical terms what the program should be 
trying to accomplish- or is in fact accomplishing. 

These are the reasons ~ some more "legitimate" than others, 
perhaps ~ why evaluations are often asked to help clarify what the 
program should be trying to accomplish, even after the program has 
been underway for some time. 

Let me make one other point about goals: So far most 
stakeholders seem to be concerned about "changing the individual 
worker" in some way. By so doing, however, programs end up 
overlooking the fact that, to improve productivity, other things (like 
equipment, decision-making procedures, institutional racism, 
compensation, and so forth) might need to change at the same time. 
Workplace programs should probably start broadening their thinking 
about their goals, to focus not solely on changing individuals but on 
changing the organization and individuals together. 

Question #2.b : What are possible indicators of 
progress toward those goals? 

As I mentioned earlier, stakeholders commonly want to know 
what the program is accomplishing. Normally they are thinking 
more specifically in terms of whether the program is meeting its 
intended (or stated) goals. Stakeholders should also be aware, of 
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course, that the program might be achieving other, "unanticipated" 
outcomes, as well, which might be useful to know about 

So there is wide agreement that, in an evaluation, it is vital to 
determine what the program is in fact accomplishing. Why is this so 
important? It's 1 ecause clear goals provide a basic standard against 
which stakeholders can judge the program's value and then make 
decisions regarding program continuation and improvement. 

Too often, however, evaluations don't get much beyond the 
"smile sheet" approach to evaluation. This term refers to the sheets 
of paper passed out to participants at the end of a course, which 
don't ask for much more feedback than whether you were happy 
with the course or not 

There is now a greater interest in clarifying in more specific 
terms what impact programs have, especially on job performance. 
This is often being done in the name of "accountability," on the 
assumption that, if we can show key decision-makers (especially 
employers) that we are having an impact on productivity, then they 
are more likely to continue the program and maybe even pay for it 
themselves. 

But good practitioners also want clearer feedback/evidence of 
what the program is accomplishing, so that they can make sure the 
program in fact meets its goals. 

To provide clearer evidence of whether a program is meeting 
its goals, it is necessary to know just what specific evidence - or 
indicators of "progress" (or "success" or "ability") - to look for. l£t 
me give you some examples of the indicators which evaluation teams 
have come up with in several different programs. 

EXAMPLES OF INDICATORS OF PROGRESS 
TOWARD JOB-RELATED GOALS 

ESL program for nurses aides (Massachusetts): 
Indicators of "oral communication" abilities included: 

• "explaining contents of trays to patients," 

• "explaining what I am doing when helping patient with 
personal care (battling, bathroom, grooming, dressing, etc.)" 
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ESL program for electoronics assembly workers (New 
Jersey): 

Indicators of "oral communications" and "teamwork" abilities 
included: 

• "explaining one's job to co-worker" 

• "asking for help" 

• "suggesting an idea" 

• "requesting a job change" 

• "giving a compliment" 

• "talking with team leader or supervisor about performance 
reviews" 

ESL program for electronics assembly workers 
(Massachusetts) : 

Indicators of job-related reading, teamwork, communication 
abilities: 

• Speaking up in meetings 

• Using technical terms 

• Reading blueprints 

• Reading safety signs 

• Reading performance appraisal forms 

• Assessing one's own goals 

• Giving feedback to co-workers 

• Writing memos 

• Writing notes to people on the next shift 

• Understanding oral communications from my supervisor 

• Understanding oral communications from my co-workers 

• Speaking to supervisors 

• Speaking to co-workers 

These are examples of specific job-related functions identified 
by planning and evaluation teams in three programs. They used 
these functions as (1) focal points for instruction and (2) focal points 
for the evaluation. Keep in mind, however, that a program might 
also choose non-job-related goals to focus on. "Helping my kids 
succeed in school" is a common non-job-related goal. In such a case, 
similar lists of indicators might be developed for each of those non- 
job-related goals. 
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Question #3: Besides determining whether goals are 
being met, what other kinds of information might an 
evaluation focus on? 

If you recall, when we dealt with our first question of 'Who 
wants to know what information for what purpose," I stated that, 
while stakeholders typically want an evaluation to clarify to what 
extent program goals are being met, that is not the only kind of 
information they want When we carefully look again at our list of 
"Types of Information Commonly Wanted by Stakeholders," we see 
that, apart from looking for information related to 
"intended/anticipated outcomes/goals," an evaluation might also look 
for information which will help stakeholders to: 

• clarify program goals(#l) 

• identify unanticipated outcomes (#2.b.) 

• clarify whether to continue "investing" in the program (#4) 

• clarify what factors are helping or hindering progress toward 
program goals (#3) . 

• decide what needs to be done (by whom and when) to improve 

the program. (#5) 

These are common types of information which stakeholders say 
they want. Practitioners tend to want to look at all of these issues. 
Funders (including employers) might focus more on the program 
continuation question. 

When you are planning the data-gathering activities you'll be 
using in an evaluation, you need not focus solely on "assessing 
progress toward stated goals. You can also^in many cases, gather 
several types of information in one data-gathering activity. For 
example, if you're interviewing a supervisor to ask what impact she 
feels the program has had, you might also ask whether the 
supervisor thinks the program should continue and, if so, how it 
might be improved. 

Question #4 : What are sources, of information? And 
hsm can information be gathered from those 
sources? 

We have now mapped out the range of information you might 
want to collect in an evaluation. The next question facing us is a dual 
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one: "What are sources of information? And how can information be 
gathered from those sources?" 

Here are some possible sources of the kinds of information 
sought in workplace education evaluations: 

POSSIBLE SOURCES OF INFORMATION 

FOR 

WORKPLACE EDUCATION EVALUATIONS 
Human sources: 

Learners 

• Current 

• Past 

Managers 

• In various departments 

• At higher levels and supervisory level 

Union representatives 
Education staff 

Employees who chose not to participate 

• Those who considered participating but chose not to. 

• Those who began participating but left the program early 
(a£.a. "dropouts"). 

Learners' co-workers 

Material sources 

Documents 

• From the workplace 

• From the classroom 

• From learner's life outside the workplace and classroom 

These are "sources" where we might find the kind of 
information our audiences want Now our question is, "what kinds 
of activities might we use to collect data from those sources?" Here 
are some ideas, drawn from experience, readings, and talking with 
other practitioners. 
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POSSIBLE DATA-GATHERING ACTIVITIES 

FOR 

WORKPLACE EDUCATION PROGRAM EVALUATION 



Interviews 

• Individuals 

• Groups ("focus groups") 



Questionnaires 

Observation (of learners, instructors, others) 

• On the job 

• In the classroom 

"Feedback sessions" (Education team members meet to review 
program activities and identify strengths, limitations, and needed 
improvements.) 

Portfolios: Folders in which individual learners (or groups of 
learners) collect evidence of their goals, interests, progress, problems. 
Contents could include: 

• interest inventories (a record of learner's evolving interests) 

• Daily/weekly tilalngllP i fn irnaTs (written notes between learner 
and instructor, or possibly between learners) 

• Writing samples 

• Attendance records 

• Rm>rt nf materials read 

• A uriin-rerordings of learner speech 

• Anecdotal observations by instructor 

Combined instructional/evaluation activities: Activities 
conducted as a normal part of the instructional process which 
facilitate learning while also providing evidence for use by instructor 
(or other person) in an evaluation. Examples include: 

• Simulations (learners demonstrate their use of particular oral 
and/or written communication skills to solve a problem in a 
simulated situation) 

• pesearrh projects (Learners identify a question they wart to 
investigate, gather relevant information, and prepare a report. 
This report can be presented orally or in writing.) 
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• Oiiira* (for vocabulary, grammar,...) 

Standardized tests (Some tests are relatively more relevant to 
particular contexts than others.) 

Review of records and sample materials 

• From the workplace 

• From the classroom 

• From learner's home life 

When considering all the possible information sources and 
data-gathering activities, you might consider combining these two 
lists in a "grid" format Here's one taken from an evaluation team 
currently implementing its own evaluation. We used it to point out 
all the possible combinations of data-sources and data-gathering 
activities. The team reviewed each option, then selected a half-dozen 
as being the most meaningful and feasible. 



POSSIBLE SOURCES OF INFORMATION 
AND WAYS OF GATHERING INFORMATION 
(Sample from a hospital education program.) 





Inter- 
view 


Focus 
Group 


Quest' 
naire 


Writ. 
Simu- 
lation 


Oral 
Simu- 
lation 


Obser- 
vation 


Doc- 

ment 

Review 


Learn- 
ers 
















Super- 
visors 
















Higher 

Level 

Mgrs. 
















Union 
Reps. 
















Instruc 
-tors 
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Tutors 



Docu- 
ments 
re: job 
perfor- 
mance 
Class- 
room 
records 



Question #5 : What ran we do to improve how we use 
evaluation? 

I have now touched on only some of the components of a solid 
evaluation, and haven't gotten into what you might do to design 
instruments and use the data once they are collected. You need to 
organize the information, summarize it, analyze it, and report it 
These steps in an evaluation process will be covered in the 
handbooks and reports which will emerge from the above-referred- 
to National Institute for Literacy and ABC CANADA projects. 

In the interest of time, I would now like to skip to the last 
question of "What can we do to improve how we use evaluation?" In 
this case, we are applying the principle of "continuous improvement" 
- an organizing principle in the workplace "quality" movement - to 
our own work in the area of evaluation. 



WHAT CAN WE DO 
TO IMPROVE HOW WE USE EVALUATION? 



1. Consider the choices a vailable to us now. 

We should examine the way programs are now approaching 

evaluation, and assess the strengths and limitations of each option: 

a. Look for The Perfect Instrument, (e.g., a "test," "a 
readability formula," "a literacy task analysis methodology") 
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- Relatively simple, "follow-the-numbers" approach to 
planning and evaluation. 

- Provides a number (e.g., a test score, a realability 
estimate) to show to hinders 

Conls: 

- Tests generally provide only one type of information, 
from a decontextualized view 

- Readability formulas don't capture the complexity of 
the actual written and oral communication tasks carried 
out in most workplaces. 

- LTAs can be useful if done well, but are only part of a 
larger planning process. 

b. Hire an Outside Evaluator. 

- Evaluator saves you effort, does most of the work for 
you. 

- "Outside experts" lend legitimacy to the evaluation and 
program 

£on!§: 

- Evaluators typically have only limited time to spend 
with you. 

- Evaluation might be seen as something controlled by 
someone else, rather than as a tool for stakeholders to 
use themselves. 

c. Squeeze evaluation in when you can. 

(The Eclectic Approach: a little of this, a little of that.) 

- You save money by doing it yourself while possibly 
getting information of use to you. 

- You get experience doing evaluation work. 
Goals: 

- You add to the burdens you already carry. 

- Information is of limited quality. 

- Information isn't used as fully as possible. 

d. Develop a systematic, "team" approach to planning 
and evaluation. Stakeholders get involved in planning and 
carrying out their own evaluation activities. They thereby: 
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• See evaluation as part of a larger planning cycle, with all 
components of program operation integrated rather than in 
isolation from each other. 

• Plan evaluations in a systematic way, attending to the 
kinds of fundamental questions discussed above. 

• Sec evaluation as not only a way of gathering information 
for decision-making, but as a way of helping the 
organization shift toward a team-based model of decision- 
making and work. 

- You integrate evaluation into a larger, coordinated 
planning strategy. 

- You produce information tailored to your needs. 

- You develop expertise. 
Con's : 

- You and other team members need to invest time and 
energy. 

- You might need help (e.g., training, technical 
assistance) to do this. 

2. Demystify evaluation. 

• See evaluation as a positive, useful tool, rather than as 
something intimidating, imposed, obtrusive. 

3. Make a personal investment. 

• Take the time to develop your expertise in evaluation. "Do 
your homework" through reading, talking with others, and 
reflective practice. 

• Argue to funders and others that good programs require good 
planning and evaluation. Make it clear that good p lannin g and 
evaluation take time, time which staff should be paid for. 
Suggest that special funding be given to evaluation-related pilot 
projects, as has been done elsewhere in the U.S. and in Canada. 

• By organizing these "investments" in evaluation and planning, 
you will not only help yourself and your program, you will also 
contribute to the "continuous improvement" of our field. 
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